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(54) Dynamic rate control scheduler for ATM networks 



(57) A Dynamic Rate Control (DRC) scheduler for 
scheduling cells for service in a generic Asynchronous 
Transfer Mode (ATM) switch is disclosed. According to 
the inventive DRC. each traffic stream associated with 
an interna! switch queue is rate-shaped according to a 
rate which consists of a minimum guaranteed rate and a 
dynamic component computed based on congestion 
information within the switch. While achieving high utili- 
zation, DRC guarantees a minimum throughput for each 
stream and fairly distributes unused bandwidth. The dis- 
tribution of unused bandwidth in DRC can be assigned 
flexibly, i.e.. the unused t)andwidth need not be shared 
in proportion to the minimum throughput guarantees, as 
in weighted fair share schedulers. Moreover, an effec- 
tive closed-loop QoS control can be built Into DRC by 
dynamically updating a set of weights based on 
observed QoS. Another salient feature of DRC is its 
ability to control congestion internal congestion at bot- 
tleneck points within a multistage switch. DRC can also 
be extended beyond the local switch in a hop-by-hop 
fashion. 
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Description ; .? s . .-. ^ ^ - • - 

BACKGROUNDrOF THE INVENTION .: : . . ^- . V r . • : \ ' - 

5 1. Field of the Invention o^-.. ' *' * , ^- . 

[0001] The present invention relates to a control scheduler for an ATM netvyork arfl, more specifically, to a scheduler 
whichguaranteesminimumrateof transmission, while fairly distributes any unused bandwidth. 1 

10 2. Description of Related Art - . - ^ • - ^ - .. .. . 

[0002] 'J High-speed networks based oa the Asynchronous Transfer Mode (ATM).are expected to carry services wifri^ a 
wide range of traffic characteristics and quality-of-service (QoS) . requirements. For example. <n audio transmissiqn a 
cell is useless to the receiver if it is delayed beyond the specified rate..pnihe other hand, video trarismission is very 

75 bursty and, lintessishaped at .the entry point, may cause temporary cpngesti^ Integrating all 

services in one network- with a uniform transpcM-l^mechanisiTi/iCan^potentialiyiS^ n^^ork ope.ration arid improve 
network efficiency In order to realize. these^potential benefits...an efficient and fair means. of /allpdating the network 
resources' is essential. '^r ' ' :* . . ■ j j ; -.f:L : - . ^--r'' -^t ' : -i-i , , . , 

[0003]- A central problem in.allocating the network resources is , the manner irp which the service to the various users 

20 is prioritized.- A sirnple model is to use a First In First Out (FIFO) algorithm, In a simple First-ln f;irst-put (FIFO) sched- 
uler, there is hb waylof guaranteeing that each: stream getSj its assigned rate. During some interval of time, a giyen 
stream may transmit at a rate higher than its. assigned rate Mj., and therjaby s^ b^rrdwicfth from other strearns which 
are transmitting at or below their assigned rates. This prablem led to the d^elopment of^vari'ous mechani 
ing the entry to the network, siich as the known:leaky bucket algorithm.. For example, the output stream for each 'queue 

25 can be peak rate shaped to a predetermined rate Mp.; , . ^ ^ ; - f ^ - - ;riev, / 

[0004] Figure 1 shows a static rate control (SRC) scheduler with N-stream queues. SQI ..SQ2 ... SON.. one queue 
corresponding to each stream. The SRC scheduler:serves a queuei at the constant rate Mi and the output cell streams^ ' 
are fed to a common bottleneck^queue CQ which is served at.a:given ratep, Service from the commori queue CO cor- 
responds to ciel I- transmission; over a link! of capacity C, c . ;*:!*)' , . - . > 

30 [0005] 'Rate-shaping transforms the streams into constant rate streams (assuming all queues are continuously tack- 
logged).* Considering the' relationship r " ^ : ' , , !7 , = . / . V ^ 

" ■ - • r ' ■ . - . \ .. : T. : N , v , ^ ^ .p - , :r. ; . • . . , - - - . . . : - - , . 

(to be developed further below)^ the bottleneck queue will be? stable; in Jact. the maximum qu^ue length is N. in , fact, 
strict inequality in (1) will usually hold, implyingihat the cell.delay in the cpmmon queue will be small with high prdba- , 
40 bility. Although the service discipline depicted in Figure 1 is work-conserving with respect to the stream queues, it is 
non-work-conserving with respect to the common queue, since jt is possible that the common queue may go empty 
even when at least one of the stream queues is non-empty. This scheduler is similar to a circuit-switched system, except 
for the asynchronous nature of the cell streams. S 

[0006] If the rates, Mj, have been computed con-ectly based on the stream traffic characteristics and QoS require- 
45 ments. the minimum rate scheduler should succeed in guaranteeing QoS for all of the streams. However, because this 
scheduler is non-wdrk-conserving with respect to the corrimpn .queue, bandwidth could be wasted for one of tv^p rea- 
sons:- " ' - ' - 'r -.1. . - . * • ^ • ^ • • - . 

• ^ The C AC algorithm was optimistic in its computation of Mj. Jt nnay be the case that a. bandwidth of Mj + A pv^r short 
50 intervals of time is required to ensure that QbS is met for Stream i, ; V. jr-r. 

• The traffic'stream could include low priority cells. with:the celljoss priority (CLP) bit set to one. . . ... 

In the'first case, a stream should be allowed to make use of bandwidth beyond its allocated rate. Mj. if the bandwidth is 
available- In the second case, the QoS guarantee applies only to'cells that conform to the negotiated traffic contract. 
55 i.e.. cells with i:ell loss priority (CLP) set to zero. However-if bandwidth is available, a stream jShould t»e permitted to 
transmit nonconforming cells, i.e., cells tagged as CLP=1.celis.,oyer and above the allocatedrminimum rate for CLP=0 
cells. If bandwidth is not available, CLP=1 cells should be.dropped before CLP=6 cells; i.e.. there should be a lower 
threshold for dropping CLP=1 cells. (As is known in the art. when a. source transmits at a.rate higher than the negotiated 
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rate, its violating cells are tagged by setting their CLP to 1.) • • 

[0007] Clearly, the problem with the minimum rate scheduler, is that streams cannot make use ^^^f^^^^.^"^^^ . 
Lven ihen it is available. In minimum rate scheduling, there is no statistical multiplexing among cells belong ng to drf-. 
ferent streams. (As is knov^n in the art, statistical multiplexing takes into account "economies of scale, '^e.. J^e band- 
wSS iUakes to\ransmit all the streams together is less than the sum of the individual '^-"f^'*^^^:^";;^^^^^ 
each stream.) A simple way to enhance this scheme is to provide means for serving a cellfrom a "^"-^"^P^ J"^"! 
whenever bandwidth^s available. During a cell time, if the common queue is emjDfy, the-schedater serv^es a cell from 

one of the non-empty stream queues. ' . \. ' ■' ^ ^ilfa^hi^n anH the P^cess 

[0008] According to another prior art method, the queue selection is done .n a round-robin fashion, and the excess 
bandwidth is shari equally among the active streams. A disadvantage of such a scheduler ,s*atqaeues are sgrv^ 
w^out Vegard to QoTTha!; is, the bandwidth is alternated sequentially to the queues vjothout ^^^^f .^.^ « ^ J^f^^^^ 
transriiission, i.e.. requested minimum rate, of any spefcific source. TMerefdre,;-th.S.method does not lend, i^ for.., 
serVinqdiherent dasseshkving different CfoS"re^uiremenfe.'->^^ •= '■ " '-'r ' ' : ' " ^ ' 

S- Accordingly there has been tonsid4Vabletntere^t^m 

weighed sha^^ of thybariii^idthbn cdnt^eti^g trafficstreaa,s; ^ ^^'f J^^J'" ^^^^ 

ent classes. With smM ^'^m ^icfess-bdhdwrdlh can-be shared usingWe.ghted Roun^Rob n (WRR)^ 
WeightedVair QuevJinb (WFQ)?ahd \r.FtuarCl6ck;^a-ndWeii=: variants/ whith attempt to approximate the idealized Gen- 
12 ProceSor.Shl;ing (GPS) scheduling, assuming a fluid model of traffic For WRa seeM. ^ateveo-s. S. Sid. 
roDbulbs and C CourCdub^etis, 'Weighted Rblahd^Ttbferi GeirMiMtipl^ atGeneral-iPurpose ATM.SwitQh, IEEE. 

jIaC vol 9 pp '^Ss-lM' b^^^^ sefe:A:K. Parekh and .R?G. GaUager.-A-genera «ed Processor 

Sh^Hng iSroach toFfdw ddnt^of in Iritegr^ Service Netwe^k^.Th^Single^lode Gase,.IEEp/ACM,,Jr^S^_a 
woA.in| V? 1 . pp-344-35r: Jtihe t993: f^r VirtWcldckf se4.€.^Zhang, .VHukI <a6ck: A*iew.Tratf.C Co0^°';^9°^*^i" 
Tpi^et SwSng.^ACM'1r^Hs! Sy^tehfei'vb^^^ p~p: 1-0t.1?4.:May.1991.ln ^t^^^^.^^^^f^^-;!^^^^^^ 

streari^is a^igried ' weight' c6rr4spohbi,ig t6 the^QdS>et,U%sf^ by «f ^'^'i^^^^^^'^r^'^^"^^^^ ^ 
of time in which the number of active streams is fixed, the bandwidth teceived by an active stream shoukLberoughly 

i^|^°^anSSi;?^^^6f^e^hi;"«^^ 

spropdrtiohal to fts weight: Hence. fe^crt>streamVeceiv«sra'iT,1nimuml.andwidth guarantee. Ita strea^^^^ 

use S^all of its guarante'«i bandwidth, the excess bandwidth tT^f^^'^'I^^^I^^^^^^ 
weights. However, a stream wrth a larger 'weight wilF hot^tihly iefcfeive a higher 

1 Jger shares of the available bandwidth than streams with smaller weights. Thus. *e.ght^"^^^""^°" 
determines not only its minimum bandwidth guarantee, but also its share of the available unus«^ . .^^ ^ 

[001 1] In this specification the term "weighted fair share scheduler" Is used generally to refer to a gene al class of 
Sionserving schedulers which schedule cells so as to give each stream a share of the link bandwidth which is 
aplx?r^Serpropo^^^ to a pre-asslgned weight. A work-conserving scheduler transmits a cell over the link when- 
Ser ZeTs af least one cell in queue Thus, a work-conserving scheduler basically determines the order ,n which 
queued cells should be sen/iced. The operation of such a scheduler is described in the following. 
Fooi2] Consider an idfealized fiLiid Hodel for bach fraff« stream. L^t -W, be the weight assigned to stream i.- At,t.me. t. 
the beneralized Proce^pr Sharing (QPS) discipline serves s^^ ' • ' • ' 
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. , ', - ■ ' y - . ' .. .- '^Ri(t)=Ur4i^MC; l.e A(t), , > , •.•.„'• . ■ ■ '" , :. (2). 

Z «j ■ . . - . ■■■■■■■ 

where A(t) is the set 6f bacWoggedstreams at time t- Thus; *ach stream always-recelves.a share of ^l^^^'^f;;^^^^ 
width which is proportional to its weight. Because of the discrete nature of cells or packets, a real ^c^jiuler can oi^y 
approximate GPS scheduling. PGPS (Packet-by packet Generalized Processor Sharing), also l^own ^^^^^^^J^ 
Queu^g (WFQ) rioted above, and Its variants (cf. S.J. Gdl^stani.-A Self-Clocked. Fair Queuing Scheme for Broadband 
SpSons ° IEEE INFOCOM 94. Toronto. June l994^.andJ. C R-Bennett-and H. Zhang. WF2Q:. Wo^.-Case Fair 
WeigS Fair Queuing, in IEEE iKlFdCOM -96, San Franclsto, ppc 1,20.128, March 1996) are schedulers which 
appLimate GPS for fScket scheduling. Other examples of scheduling schemes which attempt to a^ie/e fair sharing 
afe IS VirtualClbck a^d i«eighted- Round- Robin rioted above.- Several other weighted fair share schedulers have been, 

iJs^houS b?Spreciated that the assigned 'weighVandthe current usage.of the network would determi^^^^ . 
Ser a QoS requestS^by ari incomihg^call can be ^uar^nfeed -Therefore,. various Connection AJm^^'°" g*" "^^ 
(CAC) Algorithms We been developed which dedineservice When the QoS canlnot be gua^^ 

CAC algorithm must be able to predict the load on the system, including the newly received call if admitted. Therefore. 
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delay'bo'unds have'been found for WRR, WFQ/Vinual Clock and other fair share packeit scheduling algorithms. Using , 
these delay bounds; adniission control schemed cari be devised to provide worstfcase delay guarantees. The delay - 
bound's are ts?picait'y obfained by assuming worst-case- behavior for streams controlled by leaky bucket-type open-lopp 
flow cxbntrol mebhahisms." However, a problem in such an algorithm is that the calculated bounds terxl to be rather l<x)se.- 

5 since worst-case determiriistic'assumptiOns are made In obtain - 

[0014] Another probjeiri with the prior ^rt schedulers is as follows. Conventionally, schedulers have been designed so 
that they are vyork-conserving with respect to the stream queues, in the sense that whenever link bandwidth is available 
and a packet is in the queue, a pacRet will be transmitted over the link. In other words, if a packet is available for trans- 
mission and there is sufficient bandwidth, the packet will be transmitted and the scheduler will not be idle. The work- 

10 conserving approach has beeri promoted in the prior art sfhce it presumably results in the highest possible utilization 
overtrielink. ' • ^c -r . - • . .C' • . ^ . ?r . . * ■ 

[0015] However, within a Witching system or the hetwdrk; thiefe may be several bottlenecks. For example, some of 
the streams may be'b6ttienecl<ed at^a' downstream link ait another stage^within the^tch or the network. In this case, 
providing these streams nnore bandwidth than their minimum guaranteed rates (when bandwidth is available) could 

15 exacert>ate'the cc»igestl5n at the downstream bbttleneck.^Such congestion cannot be alleviated by the^prior art sched- 
ulers because they are wo>k-conservihg with respedt to a s^^ servicing cells only in accordance with.the 
available bandwidth at this bottleneck. That is, conventional weighted fair share schedulers always ensure that excess 
bandwidth is utilized and that the share of excess bandwidth made available to each queue is proportional to its weight, 
but tiiey do no exercise control on the absolute value of the rate received at a bottleneck point, 

20 [0016] Additionally, if there is a downstream bottleneck, typically backpressure signals which throttle upstream traffic 
are used to alleviate congestion. However, backpressure signals are susceptible to on/off oscillations, resulting in 
higher cell delay variation (CDV) and, more significantly, loss of throughput due to the equalization of bandwidtii distri- 
bution. That is. in the prior art when a buffer reabhes its linriiti a bfichqpressure sigrial is sent to tiie source. Upon receiving 
the signal the source wouid stop transmission- until the buffer signals that the pr^ssurefwas relieved. However, at that 

25 time it is likely that all the sources would start transmission aigain concurrently, thereby overloading the buffer again so 
that the backpressure signal is again generated.^Therefore. the systein may oscillate for sometime causing, large vari- 
ation jn cell delay. Additionally, since alt the sources would stop and start transmission at the same time, the throughput 
would^e equalized irrespective of'tine QoS requestki by ea 

[001 7] Since weighted faii- share "schedulers schedule cells Only with respect to a single bottleneck, throughput for a 

30 cell stream may suffer because of backpressure resulting froni downstream cbrigestion. Hence, it may not be possible 
to guarantee a minimum throughpu^^ this case' Cohsequerltly. while the' prior art weighted share scheduler is work- 
coriserving with r^pect to a bottleneck link, it may be non-work-cbnserving with respect to a downstream bottieneck. 
Thus, the present inventors have determined that work-conservation is not always a desirable property and may lead 
to further congisstion downstream. ' ' " . rv.v_ - . v ^ t- 

35 [001 8] Yet another problem with the prior art weighted fair scheduling Is tiiat they necessitate an algorithm for- search- 
ing and'sprting put the timestamps applied to the cells in order to determine the next queue to service. More specif ically, - 
in the prior art the tihestamps are relative, i.e.. "th¥sfcheduler~heeds1o'c^ the cells according.to tiieir 

timestamp. For example, the scheduler nnay order the cells according to the length of the timestamp or according to the 
time remaining before the cell would be discarded. Such calculations may slow down the scheduler. 

40 [0019] As is known in the art, the ATM Forum has established four main classes of traffic, generally divided into real 
time traffic and non-real time traffic. Constant Bit Rate (CBR) and Variable Bit Rate (VBR) are used for real time traffic, 
e.g., audio and video. Available Bit Rate (ABR) and Unspecified Bit Rate (UBR) are non-real time traffic, and are mainly 
used for computer communication. As can be appreciated. ABR traffic has no minimum rate requirement and the main 
goal in scheduling ABR cells is to "purnp" as many cells as'jpbssible using the available bit rate. 

45 [0020] A dual proportional-derivative (PD) controller for ABR service has" been proposed in A Koiarov and G. Rama- 
murthy. Design bf a Closed Loop Feed Back Contirol fbr ABR Service, in Proc.' IEEE INFOCOM :97. Kobe Japan. April 
1997. The scheduler is implemented on a hetwork-wide basis using resource management (RM) cells. Generally, the 
source generates RMCs which propagate through the network. As each RMG cell passes through a switch, it is updated 
to indicate the supportable rate. i.e.. the rate tiie source shbuld't-ansmit the data (generally called .explicit rate). These 

50 RMC cells are fed back to tiie source so tinat the source' may-adjust its transmission rate accordingly. However, the prop- 
agation of the RMCs through the network caijses a large dblay in controlling the source. While such delay is acceptable 
for scheduling ABR cells, it is unacceptable for scheduling real time traffic. Moreover, the delayneed to be accounted 
forby the scheduler, which complicates the computations arid slows down the scheduler. . - 

55 SUMMARY OF THE INVENTION ^' 

[002,1] The present invention provides a new scheduling scheme which uses statistical approaches to admission con- 
trol so as to provide much higher utilizations, while maintairiing the guaranteed QoS. The general concept of the present 
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invention is to construct the rate .from two -components. Herein, the two components, will be callecj firet.^nd second 
schedtile factors. In this event, the first schedule factor is a fixed . factor determined for -each queue^ while the seqond , 
schedule factor, a variable factor determined by a relative relationship between each queue and the ^^-^fl^Weues. 
Specifically, the first schedule factor maybe.a minimum guaranteed rate while the =f ^or^^^^^^^^^^^^" W*^^ 1^°^ • 
tion of the unused bandwidth (rate). Constructing the rate from two components «''^'?«.*'^«,f5'^H'ertPjOE^te u^^^ 
at least three modes: (1) full available. rate (i.e.. minimum guaranteed rate plus a portion of. the uriused bandwdth) (2) . 
minimum guaranteed rate, and (3) halt transmission (with very small probability). Imts preferred to^^;^7?2'DvS^ , 
uling scheme decouples the minimum guaranteed rate from the portion pf. unused bandwidth and .s called Dynamic 
Rate Gontrd (DRC). • •• ■■■ . • ' ■ ••. •-. ••• ^ ,u '• • • ^ 

[00221 -The DRC first distributes the bandwidth so as.to support the guaranteed QoS, i.e,. .rt supports the minimum 
guaranteed rate. Then, the DRC distributes any unused bandwidth to users, based upon a criteria which, 'n the pre- 
ferred embodiment, is independent of the minimum rate guaranteed to the upers. A notable feature of the loyer^tive DBP 
is that- it ife not necessarily work. conserving/. but .ratfter: takes into accpunrbottlenecks, downstream in. determining . 
whether to aHbCate unused tiandwidth; ^ . ■ . .. v " ■ Ji "-wiu ^ =ir^=»<»H 

[00231 - AS noted above; a.<Jisadvantage of theptipr.art , weighted, fair.shariog is that.the.ent.re,l?,andwKltb » allocated 
according to We assigned weight. However..it might.be^esirab|e.to-d,etermlne the seryice,rat^ according to: ^ . . 

■,. . Ri(t)=.M-it— 4-;-.E(t)._ ^, . ..(3) 

' > W i , " . , 

■ . _ . . : , ■}^^^^) -. f " 

where."in general, wip^Wj: ln:the.above'equation. the.minimum.rate gua 
L deddupled..This deciupling-allows the network provider.to disl^ 

minimum guaranteed rates. The.inventive D,RG scheduler- naturally decouples the minimum rate.gMarantee. fro^n^the 
ba^ldwidth allocated to a stream. Weights ean.be, assigned on a per-qlass basis by the .QAC or dynamically^yia 

a closed-loop QdS-contrcl njechanism. >. ■ ; fk' ., r. ' . - . . '■' " - ■ " 

[00241 Thus for example, for UBR it may be preferable to.assign very small ,or even zero guaranteed^ minimum, rgte. 
but to provide a^large portion of the availab|e<b.andwidth..This will help satisfy many real-time calls,.while provi?ling serv- 
ice for" non-real time UBR when thereis bandwidthavailabte. : • ' . . . 
[0025] Also rioted above is that since work-conserving schedulers transmit a cell whenever there is at leest one cen 
n a queue, it only determines the order in which queued pells should be ^^^''^^'^V^'^^^^^v^ r^''^*'"/,'^^^^^ 
schiuler may anow a cell time on thelink to go idle;eyen inhere are cells in queue. ^Hence.jn addition tp^tfieorde^ng 
of cells for service, timing is also important in non-work-conserving schedulers. Therefore. the present 'rjventp s h^ve 
developed mechanisms to account.for both p,«ieri.ng.and,timing.pf cell transmissiori- »^°W\'(.?'!'^^^V^^n'^',^Jr-'^J^ 
preserrt invention the timestampare absol.ute.Tattier,Wa,n.r.el,atiye. T^^^ 

a timestamp which equals the current..time is.eligible for service. Therejpre. there is no need for constat ordering of the 
cells according to timestamps.- . ,> ..; , .t . -.-n. ■ .• • • •: = / ■ • 



40 BRIEF DESCRIPTION OF THE DRAWIIVIGS , . ■ . • r , ; ■ - ' 

[0026] ... ■ ■ v . :.:•-'.. •• • ■ -1 . -.^ • ■ 

Figure 1 is a schematic illustrating a static rate control s,cli,ed,uier acc.ordiQg fo the prior" art. ...... 

Figure 2 is a schematic illustrating the.dynamic rate ,coritrol according to the present invention. _ 
Figure 3 is a blockdiagfam.depicting an:embodirT>en,t of an jnv.entive controller.^sed on matching target ut. izatioa 
> Figure 4 is a block diagram-depicting an embcdi.jient of,an Jnyenti^ye controller based on ma.tching target queue 

lengtii. . v •. • • ■ ■•. -. -.- , . .-• 

Figure 5 is a schematic illustrating theinvertiveDpC, scheduling, with .py^^^^^ . 
^ Figure 6 is a schematic illustrating tlieJnyentive DRC^scheduljng with multiple bottlenecks^ 
Figure 7 Is a schematic illustrating a rate-shaping scheduler structure for per class queuing. . 
Figure 8 is a schematic illustrating.a rate-shaping scheduler structure for per virtual channel .queuirig. 
Figure 9 is a schematic illustrating an input, output buffered switch. , , . , 

S5 DESCRIPTION OF THE PREFERRED EMBODIMENTS • . • . 

[0027] Ageneral aspect of the present.invention is the provision of ^ rate which includes two «'.'"P°"^"f^^^.";'|™"?^^^ 
guaranteed rate and a share cf the excess bandwidth. This ajlovys ttie scheduler to provide . service according to various 
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QoS^'requirerrients and to shape 'the rates wHilie accounting for downstream bottlenecks, ^ . . - , 

[0028] Unlike GPS-type schedulers, which distribute the entire bahdwidth according to assigned weights, the inven- 
tive scheduler first services the minimum rate and then may or may not distribute the unused bandwidth. In its simpler 
version, this approach can be derived as follows. - 

, which is a function of the entire 



5 [0029] ' A fairshare scheduler p|-6vides stream I with a* minimum bandwidth guarantee, 
available bandwidth, ile..' ■ >^ - r . 



10 



Clearly. 

15 



. J;.. Mj = :^G. - . ^ : . . , /• (4) 



[0030] However/ it is preferable f^ the rate into' tWo components: the share of the bandwidth as provided by 

the minimum rktei plus thi^'dhare'of the bandwidth. When A{ty is thS set ©tractive streanris, the two components 

rate can be written asi ' * ' - ^ "* ^~ ' • ^ - .:. ^ ^; > . . - 
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■RiO)-Tr^c + w,.^i^^flw.c.,. , ,. . . , . .. ..(6)- 

• ^ • j=1 - ■■■ ■ ■ JiV jeAp)- ■' •• ^ . ■: 



where 



'IN ^ 

j=1 _ _ je;A(t) * ■ 



= Mi + — ^''E(t)ri G'A(t) ' * ■ * - (7) 
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= C- XMj. : ' ^^^^ 

In Equation (8). E(t) is representative of the excess or the unused bandwidth available at time t. Referring to Eq. (6), we 
see that the rate at which stream i is served is the sum of the minimum guaranteed rate Mj and a weighted fraction of 
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the unused bandwidth E(t). Thus, depending on theilpad on downstream bufters, the,scheduler may ,or may not u^e.^e- . 
second component, i.e.. may or may not distribute the. unused bandwidth. Ijqwever. it rpay.always continue, toprov.de, 
the guaranteed minimum rate. , ■ • ^ -., • .. .- . . • . • , •. , m ; 

[0031 1 While the above scheduler is capable of guaranteeing minimum rate-and shaping the rate rising the exceg 

s bandwidth, it -lacks, flexibility insofar as>the distributipr, .of the excess bandwidth-is glosely, correlated ^ : " 

weights From Eq (6). it is clear that the rate at which stream i is served at time t is the weight Wj multiplied byjiesum 
of the link capacity normalized by the sum of the weights over all streams, and the unused bandwidth normalized by the 
sum of the weights over all active streams. Hence, both the minimum guaranteed rate M; and the excess rate Ei( ) are 
proportional to W, This is not necessarily desirable from the network provider's point of view The network provider may 

10 ' prefer to distribute the unused bandwidth in a different proportion than that of the minimum guaranteed rates M, 

[0032] Therefore, unless othen^ise noted, the remaining description refers to the prefen-ed embodiment of the present 
invention wherein a novel dynamic rate control (DRC) is provided which ensures the guaranteed QoS and d'^ributes 
' unused bandwidth in an efficient manner decoupled from the minimum guaranteed rate. The inventive DRC s not nec- 
essarily work consenting, but rather takes into account bottlenecks downstream in determining whether to allocate^ 

looSf 'inihe cSe of a single bottleneck link shared by a set of traffic streams, the inventive DRC provides a minimum 
ate guarantee for each stream. Streams which do not make ful.l use of their minimum rate guarantees (i.e., streams with 
input rates less than their minimum rate guarantees) contribute to a pool of excess bandwidth which is "«de avaHaWe 
to streams which transmit in excess of their minimum rates. In DRC scheduling, the distribution of he bandwidth 
is determined by weights assigned to the streams. In contrast with weighted fair share schedulers, the share of the 
excess bandwidth which is made, available to a stream i^.the inventive DRC is.decpuplec^.fron, the. rrnnimum rate guar- 
antees; i.e. the share of the unused band-A/idth need not.be proportional to the.assigned.niin.muoi ra^ guara^ . 
f00341 The DRC scheme also strives to provide the minimum rate guarantee on a short tme-scale-.That/s, the pHO 
scheduler paces the cells of each stream queue such that the spacing between cells Ijelonging to the same s^reamjs 
no smaller than the reciprocal of the minimum rate. If the connection admission control "^f /'"'"^^^f.f ^'^ .^"^^^^^ 
bandwidth requirement fora streamto meetagiven QpS, the DRC scheduler should b« f'e o deliver me wed^C^oS 
by virtue of te ability to guarantee this minimum raterMorepven the DRC scheduler distnbutes unused bandwidth in a 
fair manner among the competing traffic streams. V , ^ / . nntontiai hnt- 

[0035] When there are multiple bottlenecks in a swftch. the DRC scheme can eliminate congestion a* all poten^aJ bot 
tienecks for a given traffic stream. In contrast to the prior art weighted share ^^^edi^ers the invertive D RC^^^^^^^^ 
minimum rate guarantees even in the presence of multiple bottlenecks along a path within the switch. When there are 
multiple bottlenecks, the share of unused bandwidth givea-to a- stream at-a given bottleneck may also ^epend oj the 
state of the downstream bottlenecks. In this case, rate feedback from each bottleneck encountered by a stream is used 
to choose the maximum rate at which a virtual channel (VC) can send without causing congestion ^urthej^more DRC 
can be extended beyond the switch in a hop-by-hop flow control scheme which can provide JnH !.L1n" 

tees (As is known in the art, the term virtual channel refers to a link of communication which is established and. main- 
tained for the duration of each cell. The link is called.virtual.cbannel.since, unlike synchronous transmission, there is no 
set channel designated to a particular caller.) ■„ .v,^ ^r,rr,n,^=i\r,n 

[0036] In the iiwentive DRC scheme, the excess bandwidth is shared amongst competing users via the computet on 
of dynamic rates in a closed-loop feedback loop. DRC scheduling also requires the internal transmission o control |nfor. 
mation within the switch. Notably, the DRC scheme lends itself to a relatively simple ^ate-shaping scheduler imp^me^^ 
tation. Unlike fair share schedulers based on timestamps, no searching or sorting is required to find the smallest 
timestamp. 

[0037] The main features of the DRC scheduler are outlined below: 
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1. Provides minimum rate guarantees for each stream: - , - . - w ^ * • ^ u 

2. Allows flexible distribution of excess bandwidth. The .?hare of excess bandwidth can be determined by: 

(a) Static weights according to traffic class (or other criteria) set by the connection admission control (C AC) and 
may be called secondary or class weights. Each cl^ss weight may be multiplied by the number of active virtual 

' . channels (VCs) belonging to the given class to achieve fairness with respect to VCs. 

(b) Dynamic weights determined according to observed quality-of-service (QoS) by a dynamic closed-loop 
control mechanism. 

3. Provides internal switch congestion control. This is advantageous especially for providing minimum rate guaran- 
tees without overflow of buffers. -1. U ^.^ntr^l 

. 4. Allows extensibility to provide minimum rate .guarantees on an endrto-end basis yia hgp-by-hop flow control. 
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Dynamic Rate Control Principle 

[0038] This section describes the principles behind the inventive dynamic rate control scheduling. Consider a set of 
N ATM cell streams multiplexed onto a link of capacity C. Each stream may correspond to a single virtual connection 
5 (VC). or a group of VCs belonging to the same traffic class, i.e:, a group of VCs requiring the same QoS. Associated 
with each stream is a queue which stores cells waiting for service, i.e., cells waiting to be transmitted over the link. The 
function of a scheduler is to determine when the queued cells are to be serviced. 

[0039] As noted above, the inventive DRC ensures the guaranteed QoS. In the preferred embodiment of the inventive 
DRC scheduling, the QoS guarantees are mapped onto minimum rate guarantees; however, it should be understood 

10 that other mappings may be used. That is. according to the preferred embodiment, the traffic characteristics (obtained 
from some combination of traffic descriptors and traffic measurement) and QoS requirements for stream i are mapped 
onto a rate. Mi. which is to be provided by the DRC scheduler. If the mapping is done correctly, guaranteeing the rate 
Mj is then tantamount to providing the QoS guarantee. It is therefore imperative for the scheduler according to the pre- 
ferred embodiment to be able to guarantee the minimum rate Mj for each stream. 

15 [0040] Fore the purposes of this discussion, it is useful to think of stream i as a group of VCs belonging to the same 
traffic class, with the same QoS requirements. An embodiment of a per VC.queuing will be discussed in a later section. 
For a given traffic stream i. the bandwidth Mj, required to meet cell loss and delay requirements can be computed based 
on the traffic parameter^ of the individual VC^ and the buffer size. The multiclass^connectiori admission control (CAC) 
scheme developed in G.-RahTamurtfiy arid Q. Ren. Multi -CI ass. Connection Admission Control Policy for High Speed 

20 ATM Switches, in Proc: IEEE INF0COM -'97, Kobe. Japan. April 1997, provides procedures for computing Mj for CBR,. 
VBR, and ABR traffic classes based on the traffic parameters- declared by individual VCs. The CAC described in that 
paper takes irito accbunt statistical rnultiplexing^aih when there, are rriany .VCs belonging, to a stream^nd can further 
be made more aggressive m its allbcatioris by ihcorporatingiraffie measurements at the switch. Given the rate Mj fpr 
each stream; the most irnpbrtant requii-ement of a scheduter is to ensure that each stream receives service at rate Mj. 

25 For stability of the systern, clearly we must have the e^qiiation 1 noted above hold true, i.e., the sum of all the individual, 
rates must be less or equal to the rate of the common queue. ' ! ' ' ' - • ' 

[0041] In developing the theory behind the inventive Dynamic Rate Control (DRC) scheme, an idealized fluid nxxJel 
for the scheduler is assumed. The stream queues are simultaneously served at the dynamic rates R| as fluid flows. The 
input streams to the stream queues consist of discrete cells. Each cell brings a batch of work to the stream queue at 

30 which it arrives, the abtual implementation' of the DRC scheme according to the preferred embodiment is an approxi: 
mation to the idealized model- ' " - ^ ^ ,1 r . , : 

Continuous-time Model : ^ . i , ^. 

35 [0042] Let A(t) be the set of active streams at time t. A streamis considered active, if its corresponding stream queue 
is backlogged. The most general form of the dynamic rate associated with stream i is given by 

Ri(t) = Mj + 4»i{t)E(t). • (11) 

40 where M| is a minimum guaranteed rate, E(t) is the excess rate^available to all streams at the common bottleneck, and 
<l>i(t) e [0,1] is a normalized weighing factor. That is, the dynam[crate comprises two components: the guaranteed min- 
imum rate, Mj, and a part of the unused bandwidth, E(t), determine according to the weighing factor. (t>i(t). assigned to 
that stream. Define 

45 \ ; ■'■ - * E((t) = (t>i(t)E(t). . : - ; r. .. .(12) 

as the variable component of the DRC rate for stream i. The excess rate is defined by: 

E(t) = C v -X ^j- ^^^^ 

' jeA{t) . ■ 

wherein C is the rate of the common queue and Mj is the actual transmission rate of stream j of the set A(t) of active 
streams at tirhe t. The'weights 4>j(t). i e^'A(t) refled how theexcess bandwidth is to be allocated among the streams and 
55 are normalized such that : - ^ • . . 
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Z 4>i(t) = i- 



(14) 



ieA(t) 



The' weights- (l>i(t) are normalized versions of positive weights Wj{t)-, i e. A(t), 
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(15). 



othe,rwise 



Eqs. (11) and (13)' define an idealized DRCsclitedulipg scheme tor the fluid oipdel.. The, basic cpnc^pt of the DRC, 

SSS* irSlS^^n'piS'Jfeach.stream t^^^^ oi - QNl^-eachlof whi<ih being served dynamically yar, 

iabie rate R1 - RN.' Each dynamic rate Ri is made of the guaranteed minimunirate Mi an^ a.share. of the ^n^ 

width Ei, termed the DRC rate. The flow from alWhe queues is,fed to the common queue CQ.^^^^^ 

[00441 In practice, it is very difficult to track the sbt function A(t).. since it can change:wi«?highftequenpy. Hence, ,t is 

mpractical to computeiHe unused fc-andWidthvvia.Eq. ()13H.lf.the set.of traffic,. strean?s-i^..rea?9nal?ly. large th£ co": 

tributionfrdmanindi^«dua^stream1SSmall:telativetothe)aggr«^ates 

Under these conditionsf the rSte af which.-E(t)Tchanges-by avsignificant amount should be. rrvuqh ^^^^^^^f^?^-"^^. . JtJ 
which A(t) dhanges. This sug&ests.that-it.1s more.feasible to track.E(t) directly, than to t^ack.A t) an^then. compute E(t) 
via Eq (13) In DRC. E(t) is estimated using a feedback control. loop to. be discussed more fully below. , 



Discrete-time Row Model 

[00451 -It is instructive td Consider a discrete-timerroodelewber.e ticneJs par-titioned irTto.interva!.s,of length A Assume., 
[^strelr: I arrives to its associated stream queue as a constant flud flow of rate JCKn) "^^^^^^^J^: 
Tn = (nA (n + 1) A) . In this time interval, stream queue i is served at the constant rate R i(n) = M , + W iE(n) , where 
is a fixed weight assigned to stream i. The output flow rate. Fi{n). from queue i during the interval !„ is then, given by . 

■ .B;(n> x.-minfR7(n). )ti(n));: , , -. . i i S^^^ 

That is. if the queue is backlogged. the rate would bVRi(n);"otheiwise it would be the arrival rate ^((n). The aggregate 
flow rate to the txjttleneck queue during Tn is then '^iili*; < 



40 



■•F(n)'*.2^F,(n)r 



(17) 



.5 The excess bandwidth, E(n). over the interval T„ is the sum of a static, unallocated portion of the band««dth^ and^^^ 
dynamic part of the bandwidth that is currently not used by streams which transmit at less than the.r minimum guaran 

teed rates: ' ' •■ •'■^ " ' ' ' 



50 



(18) 



where x^A max (x.O). Again; since it "is difficult to obtain.kriowledge of the input flow rates Xi(n). the present inventors 
55 developed an indirect means of computing E(n) via a control loop. 
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Closed-loop Control ' ^ y.::^-' ^- - - \. ' v:, ■ h. ■ ■ ■ ; ■ i 

[0046] The exceis bandwidth E cah be estirriated via a feedback control loop. Byadjustingithe value of- E, the aggre- 
gate flow rate to the bottleneck queue, denoted by F, can be controlled such that: 

1 . The length of the bottleneck queue is close to a target value Qq; or 

2. The average utilization at the bottleneck queue is close to a target utilization value Uq < ^1 • . 

[0047] Two control algorithms are disclosed herein forestimating E based, respectively, on matching a target queue 
10 length and a target utilization. Also disclosed herein is a hybrid control algorithm which combines the merits of the first 
two. 

Matching a Target Utilization ^ i ^ • . . 

15 [004iB] Consider the continuous-time model of the 'scHeduler. Let T{t} denote the aggregate flow rate into the bottle: 
neck queue at tinie t. We wish to cohtr^ol F(t) to achieve a target utilization tJo *^ (0-^)-* • ■ - • , • ' ^ 

[0049] The following proportional control law can be used: r ^ ^ *i . • > 

F(t) = ao(F(t)-UoC). (19) 

20 - 

[0050] That is. the rate of change of the aggregate flow rate is proportioned to the aggregate f bw rate less than prod- 
uct of the target utilization and the rate of the common queue y - • . * : 
[0051] The control system is stable for oq > 0 converges with exponential decay rate ao- In terms of the input streams, 
the aggregate flow rate can be expressed as: , ^ ■ > - t ' . ^) 
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= ' X Mj + E(t) (21) 



[0052] Taking derivatives on both sides (wherever F(t) is differentiable), we have 

35 ' • • ■\' F(t)v E(t);' ' :\ ;^ . . (22) 

Hence, th^ control law (19) carl be re-Wi^^ ^^r,- : ' - r; . , ; ^ 

; - ^E(t) = a5[F(r^UoC]-'-^;^-- - .-^ - ^ - (23): 



[0053] The control law Eq. (23) is the basis for a method for estimating E(t) In the discrete-time model. The discrete- 
time form of Eq. (23). is as follows: 

_ . E(n+1) = E(n).ao e (n). (24) 

where we define the error signal as G*(n) = F(h) - UgC: Since the excess bandwidth must lie in the interval [O.C]. the 
control javy takes the form: * ' - - 

- , " / / E(ri + 1)=lf(i;ci(E(nF^ ; - ■ ^ {2S) 



where I ^ qj(x) = 1 if (x) equals or larger than zero, but equal or less than C; othenwise. 1 ^ ^^(x) = 0 . Over an interval 
in which the input fluid stream flows are' coristant. the recursion in Eq. (25)' wiir converge to the correct value for the 
excess bandwidth E. The speed of convergence depends on the values of the coefficient oq and the sampling interval A 
[0054] Figure 3 shows a block diagram of the controller based on matching the target utilization Uq. The error is cal- 
55 culated by adder 10 and Is provided to the controller 20. The controller 20 outputs the current excess bandwidth E(n) 
which is fed back to the DRC scheduler 30. In practice, there is a delay in the feedback loop between the controller and 
the scheduler. However, within a switch, this delay y is typically negligible relative to the sampling interval A and can be 
ignored. The DRC scheduler allocates the excess bandwidth E(n) to the input streams (Xi(n) - Xn(n) according to the 
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DRC scheduler disclosed above, which results in an aggregate flow rate F(n). Matching queue length ■ 
[0055] Let Q(t) be the length of the bottleneck queue at time t and let Qo be a target queue length value. Assuming a 
fluid model of trafticjhe queue length growre according to the-aggregate rate less the commonque^^^^^^^^^ . . • 

Q(t)='F(t)-C. "'" (^6) 

The proportional control law. - > • ; ■ 

:, , ., ... -.. .F.(t)=--a:o(Q(t)-Qo).- - . t . • • • . % 

leads to the equation - 

Q(t) + aoQ(t)=a'o. . 

(See e g U Eenmohamed and S. Meerkov. Feedback Control of .Congestion, in Packet Switching Netyyork?: The Case 
of a Single Congested Node. IEEE/ACM Trans, ori Netwgrking. vpl, 1 . pp. 693-708. December 1.993.) Th^ characterjstjc 
equation for (28) has a double root, implying non-decaying, oscillatory behavior of,Q(t), This.problem.c^n.be resolved 
by adding a derivative term to (27). resulting in the proportional-derivative (PD) controller 

F(t) = -a'o(Q(t)-Qo)-«'iQ(t) ^^^^ 
The con^esponding differential equation governing the behavipi; of Q(t) is: . , „ ■ , .:::-..-„•, s , 

' " ." " ci(t) + a-iQ(t) + aoQ(t) = a'oQo. .. ' - ' - " ' . ■ 

which is stable if a'o. a, > 0. The convergence rate can he assigned ,3rt3itrarily by appropriate choices for o'o and a\ . 
From (29) and (21 ), the unused bandwidth can be obtained as: 

E(t) = -ao(Q(t):Qo)-a:iQ(t) 

Then a discrete-time controller can be obtained from (3 1 ) as 

E(n+1) = ![o.c](EW -.aVe (n)-.aV6..(n-1)). ;■ . , . „ ..-.■.. (32) 

where e(n) = Q(n)-Qo is the error signal sampled at time^n: Here. Q(n) is the queue length sampled at time t- nA . 
This controller attempts to keep the queue length near Qo, maintaining the utilization at 100%. 
[00561 Figure 4 shows a block diagram of the controller. The target queue length po is suljtracted from the^ qu^e 
length at time n. Q(n). by the adder 14, to provide the error E(n) to the controller 24. The controller 24 outputs the 
united bandwidth E(n) to be feedback to the DRC controller. Agajn.a delay y may be introduced, but may be ignored 
when the DRC is uses within the switch. The DRC controller then allocates the available bandwidth to generate an 
aggregate flow rate. F(n). - •' . : - -i ■■• 

Hybrid Cqntrol 

[0057] Clearly the disadvantage of matching the flow rate to a "target utilization, Uq. is that bandwidth Is lostsince Uq 
must be less than one. However, if que.ue.length inforniatiqr, is not readily available, the control algprithQi bas«J on flovv 
rate measurement is a viable alternative. The control algorithm based on queue length information achieves. 100 ^ uti- 
lization A disadvantage of this algorithm, however, is that when the utilization is less than 1 00% the system 's not con- 
trolled. It the utilization is less than 100%, the queue length Is.zero; hence. E(n) reaches the maximum value C Now if 
the aggregate traffic flow increases to a rate close to C, the queue'may grow to a large value before the controller can 
bring the queue length to.the target value Qq. ... ...... ; . %u,j.„^,„Ka 

[0058] .If both flow rate and queue length information are available, the merits of both controller algorithms, may be 
combined in a hybrid controller as follows: • .- . . ^ • - , . • • ■ ■ . . - - 
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' ifF(ri)<UoCthen - - ^ • - - ' - 

E(n+1) = E(n) - ao(F(n) - UqC) - - - 

■ ^ - - : -^ise^'-- < '".t .... . "■ . . 'T- .. . 

, .^...E(n+l) = E(nX-a'o(Q^n}-Qo>-a'^^^^ / 

end if • ■ -^-^ ■ ... - ■ . . -■ V .-s 

/5 We remark that the coefficients oq, ai. will be different for each of the two controllers. Thus, when the utilization is less 
than Uo < 1 ,-the utilizationrbased controller attenipts to nratch the^terget utilization. When the utilization exceeds Uq, the 
hybrid controller switches to the queue-based controller. This results in a more controlled queue growth as the aggre- 
gate traffic flow increases. .- '■■J '= ' 

[0059] As can be seen, while the underlying principle of the hybrid controller is somewhat similar to the dual propor- 
20 tional-derivative (PD) controller proposed in the Kolarov.and Ramamurthy paper cited above, the present invention ena- 
bles use of the PD controller for real time traffic. This is made possible, among others, by implementing the PD 
controller on a switch, rather than a network-wide basis. Consequently,- the delay of the RMCs becomes negligible so 
that RI^Cs can be used to control real time traffic. Additionally, the practical elimination of the delay allows simplification 
of the calculation' and speed-ijp of the c»rrtroller- : ' ": j ;r ^ - • . . . , 

25 " 

Overload Control ' ' ^ * . ' - * - 

[0060] The hybrid closed-loop controller adjusts the unused bandwidth E(n) ,to reduce the magnitude of an error signal 

e (n), =. < . ,j (33)' 

' • [ Q(n) '- 'Uo ''Otherwise - ■ > . >^ - 

35 However, the dynamics of the aggregate stream may be faster than that of the closed-loop controller.<The queue length. 
Q(n), may stilt grow to a large value before the closed-loop controller can bring it close to the target value Qq. The queue 
growth' is caused by streafiis which transmit at rates 'largerithan> their minimum ^aranteed. rates. Thjs rnay resutt in 
buffer overflow in the second stage queue and/or unacceptable delays for streams which are transmitting at close to - 
their minimum rates. Depending oh the value of A. the response time of the closed-loop controller may be too slow to . 

40 prevent buffer overload in the second sfege queue. - \s:rj: , i . ■ 

[0061] Therefore, in the preferred embodiment an overload control mechanism is employed which reacts quickly, to 
control congestion. DRC scheduling with overload control is illustrated in Figure 5. When the bottleneck queue length 
exceeds the shape threshold Qi > Qq. a shape signal is transmitted to the DRC scheduler. The: PRC scheduler 
responds by sihaping all streams to their minimum rates. This is equivalent to driving the unused bandwidth signal, E(n), 

45 to zero: The tirrie for the shape signal to propagate to the DRC scheduler and take effect should be much smaller than 
the DRC sampling interval A. When the queue level falls below Q-i. the DRC controller reverts back to the hybrid con- 
troller '' '■ ' * • • . . r - ^ , - , , . . , 

[0062] To prevent buffer overflow in the second stage queuei, a stop signal can be used. When the second stage ^ 
queue length exceeds a stop threshold Q2 > Qi . a stop signal is transmitted to the DRC scheduler. In this case, the DRC ' 
50 scheduler completely throttles all traff ic streams until the queue length falls below Qg. This control mechanism imposes • 
a maximum length for the second stage queue. In practice, the stop signal should be activated with low probability. This - 
will be the case if the thresholds and are chosen correctly. With the shape and stop overload control signals, the 
scheduling rate for each stream at the nth sampling interval can be expressed as: 



55 



/?,(/2)=lA/.-h7(^;^,)(^Cn)r)^ ' (34): 
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where lx(x) denotes the indicator function on the set X. Wherein l,Q<Qi)(Uln) ) = "> '< Q(") < Q"*- ^"^ 
I ,Q,Q,){Q(n) ) = 0 othen^ise. Similarly. I ,Q<Q2)(Q(n) ) = 1 if Q(n) < Q2, arid, I ,Q<Q2).(P(n) ) = 0 otherwise. 

Multiple Bottlenecks . . • ' ; .- . ■ ■ 1 

* [0063] One notable application for the inventive DRC is in a multi stage ATM switch yyhich has been already filed by 
the same applicant. In a multi-stage ATM switch, there are multiple bottleneck points: A stream may P^ss through sev- 
eral stages before reaching an output line of the switch. In the DRC approach, the streams are rate-controlled at me 
input stage to control congestion within the switch itself, as opposed to controlling flow in the network Therefore, the 

10' bulk of the cell buffering occurs at.the input stage of the svyitch, .. , ^„ c:(i)/„> io 

[0064] Consider an individual stream which passes through B botflenetks. At the j-th bottleneck, a DRC rate E"'(n) is 
computed at the nth sampling interval. Define the overall bottleneck excess rate as: 

E * (n) = minE®(n) iSjsB (35) 

'* Let-Q,(D aixJ Qz") denote, respectively, tt^e^ sfiajijfe^^nd stop fhreSHOlds at thejth bottleneck Define*e:vectors: 

Q, = IQ|^>:1 ^j^B], 1 = 1.2. . ' (36) < 

20 [0065] Denote the queue fength at the jthbottleneckratrtimeti by Q(jHn) and def m 

Then, in analogy to Eq. (34). the dynamic rate for stream i for the multiple bottleneck case is .computed as: 

Ri(n)=[M,+ l,Q^Qi,(Q(n)) • WiE'(n)] • l,Q,Q2,(Q(n)). . (38): 

[0066] «• Figure 6 shows a set of stream queues. Q1 .vQN; aloug with a.set,of bottlenepk queM,es..At,the i.th bottleneck 
queue a DRC rate. E, is estimated based on flow and queue length information. For a given stream, e.g. ST3, the over- 
30 all DRC rate is the minimum of the bottleneck rates for bottlenecks traversed by the stream. From the perspective of the 
qiven stream queue, congestion in downstrkm botilenfecks ar^e controlled and the-<Mjeuing of cells is pushed upstream 
when congestion arises at one or more bottJenecl©, yitimately, the congestion is pushed back to the stream queue, 
where most of the queuing takes place for the stream. 

35 Rate-shaping-Scheduler ■ ■ 3!..c- >n.' v.n; - : . v .. ? . > .• %r 

[0067] In order to implfement.DRa a mechanism forifate-shaping a number of ,streaRns , is necessary. Two implemen- 
tations of a scheduler which shapes a set of streamS=aceordii3«^to,aRC>rates^re,disdos.ed,,herein. .The first is.a^ 
priate wheri the nOrtlbef of stfeams Is-relatively small (oatherorderrof a huncJred or l^s>. f0r example, when cells ar^,. 
40 queued according to class. The second implementation can handle a large nuijiber of streams (on the order of tens of 
thousands), butis slightly more complex. -,>f : .- < , i :- lu:- tfv. - , .• >i v . ; 

Scheduling for'Rate-ishaping :i •. o . v ; > • .j - • :-■ • - ' - ' 

45 [0068J DRC scheduliflg is tmptemented using timestampS.,A:timestamp.;TS. is. associated vyith eachqueue. -A stream 
is active if and only id-its^corresporhding queue" is. non-empty. :Qthen«ise. the ?tream Js .inact|ye. The DRC scheduler 
schedules only active streams for service. When a stream is served, the f irst cell in the associated queue is transmitted 
to ttie second stage qui^ue and the status of.Jhe stream is upd.^ed>. •'. •■' u ■ 1j 

[0069] Two distinct timestamp. computation formulas; are.provjded depending an.whether a queue to be^schedyled 

50 or rescheduled; The timestamp computations ensurelhat each.stpeam: is s.haped to the appropriate rate, as determined 
by the DRC scheme. • > ■ ' ' v ? ^r- :.■ ■ ■, , • 

Scheduling ' • • --^ ' . ■' ' ' • • " ■• "■ •' 

55 [0070] A given queue is scheduled when the queue iSjempty and a new cell arrives to the queue; i.e.. when tt>e asso- 
ciated stream changes from the inactive to the.active state. The ba^ic formula for computing the new timestamp for 
scheduling queue i is given as follows: 
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TSj = max{CT, TSi + 1/Rj(n)). . - :^{39) 

where CT is the buirehttirri^, CT G(nA(n+1)A). ahd^R^^ , 
Rescheduling - '* . '.r - \ ... . ; . 

[0071] A given queue is rescheduled after ah active stream has been served and the stream remains active, i.e., its 
associate queue i-ehn^lns non-empty. In this ease, the timestamp computation for rescheduling queue i is: - 

TSj=:TSj + 1/Rj(n). (40) 
Catching-up with current time : - = 

15 [0072] Queue i is said to be ready at current time CT if TSj < CT. This means that the queue can be served while 
conforming to its assigned dynamic rate Rj. In a practical implementation of DRC scheduling, it is possible that the.sum 
of the dynamic streams may exceed the link capacity, i.e., 

Ri(n)>c^^ :^ v ':. ... ,^ 

If this condition 'persists,' this set of ready queues rnay ^ increase and the timestanops of some ready queues -may fall 
behind th§ curVerlt time,' CT, by iSarge vailues. " - • ' : t -'^-j " * * - > . ^ 

25 [0073] To correct thi'^ situatiori,' a mechanisrh'is-provided' whereby a queue is scheduled at its minimum guaranteed 
rate if its associated timestannp falls behind curreht time by a designated amount. In particular, if the queue timestamp- 
falls biihind current timie by more than the reciprocal of the niirtimurn' guaranteed rate at the time of schedujing/resched- 
uling. the queue is scheduled/rescheduled at the rriiriimum guaranteed rate. Scheduling a queue.at its minimum.guar- 
anteed rate allows its timestamp to 'catch lip' with the current time clock, by slowing down the rate at which the queued 

30 is being scheduled. With the 'catch upi' provision; the scheduling arid rescheduling procedures.are as follows: 

Scheduling: 

if TSi<Cr. l/^ ' ■ : : 

TSi .= max{CT, TS^ + 1 / Mi} 

' else - ^ 

TSi .= max{CT, TSi + 1 / Ri(n)} 
^^^^^^ " : " • 

Rescheduling ; 
if TSi < CT - 1/Mi then . ; . 
TSj < TSi + 1/Mi 

else 

55 W:/'-, ' ' ^TSi.<cXSi+ l/Ri(n),. . _ 

" ' endif . . ■ /' . / ' \' ' , ; . * 
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Serving Ready Queues it ■ ' ■ •• ' • ■• "if r .. . 

ro074] Serving a ready queue consists of transmitting the first cell in the queue to the bottleneck queue and if nec- 
essary rescheduling the queue for service. It is possible that several queues may become ready at,tbe ,saipe time. In 
?ac ^e Jt s nIt poLible to serve all ready queues during one cell time. Thus, a collection of ready queues can form 
10075] The ready queues can in turn be scheduled for service by means of a work-conserving scheduler For exan, _ 
ready queues could simply be served In round-robin fashion. Alternatively, a weighted fa.r share schjl"ler couW 
be us^ to ensure fair short ?me-scale bandwidth share among the ready ^-^-'^ ^^^^^''^■.'"^^^ 
share does not warrant ihe considerable additional; complexity required to implement weighted fair share scheduhpg. 
[00761 The preferred embodiment implements a round-robin with four priority levels, listed below in decreasing order 
Of priority: . • ' ^ . ■ 



1 . Dynamic high priority (HP) 

2. Real-time, Short CDV (RT-S) 
15 3. Real-time, liong GDV^(RT^L) 

4: Non-real-time (NRT) « 



The HP priority is a dynamic assignment. Ready queues which have been scheduled at their minimum Q^^^^J'^ 
rati are automatically as HR This ensures that all streams receive Jheir minimum rate guarantees on a sho^ time- 
scaTe -^e remaining three priority levels are assigned statically, accf ding to traffic class and tolerance for cell delay 
SriSio^ (CDV) Str'eams dassifL as RT-S are real-time streams which have small CDV tolerances. wMe RT-L 
streams have larger CDV tolerances. Non-real-time (NRT) streams generally do not have requirements on CDV. 
[MT^ "US, low bit-rate real-time streams wculd.be classified., as RT-l,.. while. high '^j^-^^M^f "^J^^ 
LbuW be classified as RT-S. However, the CDV tolerance of a stream need not be dire^ly r,elated tP,Jg;^^^*?^^f 
Static priority levels pfbteet streams: with ^mall GDV.tolerance from the bunching effects of streams with larg^ COy^, 
;S^es. Fbrexamp^. cbhsider a scenario in which.there are one thousand75:kbps.vo,ce stream? sf^rin^a 50 r|ps 
link with a single 75 Mbps multimediastre^.mrAssurT]ing.thatthe.multimediastrearn is.constant bit (CB^) it nj^s 
to send a cell=once eve^ two ceU times. -If cells^from the voice streams arebunched together.(at or near tlie sa^me times- . 
Jotrre^uttlmSa^irwill su^^^ 

two cells from the multimedia stream could be separatedi.by;uR4o one thousand ypic^ ... 
Per Class Queuing 

[0078] In the case of per class queuing, when the numl^er of , streams is relatively small ttie scheduler can be imple- 
mented with a parallel array of comparators. The ith comparator takes as inputs CT and TS; and evaluates 

_ j ICT ^ TSi and queue, i active ^ ^^^-j 
~ [OCT < TSi 

[0079] Queues with f; = 1 have had their timestamps expire and hence are ready for service. These queues are served 
using round-robin with priority based on a priority flag Pi assigned as follows: 

45 0 if queue i scheduled at rate M; 

Pi = 1 if queue i is RT-S ' (43) 

2 if queue i is RT-L ' - " 

3 if queue i is N'^T ^ ■ ^ 
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[0080] A logical view of the scheduler is illustrated in . Figure 7. The scheduler per forms a round-robin search for a 
Queue i satisfying f = 1 and 0. If no such queue exists, the round-robin search continues by searching for queue . 
sSying fl 1 ^nd Pi = 1 The process continues until either a queue is found, or all priority levels have been searched. 
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Per VC Queuing , . ... 

[0081] In per VG queuing, the^nurtiber of queues is on the order of tens of thousands. In this case, it is not economical^ 
to implemerit th'e sdieduler usirig a parallel array^of comparators.: Instead, a scheduler based on a time wheel data. 

5 structure is preferable>As shown in Figure 8. each bin in the time wheel points to a four linked lists (one for each priority 
level) of VC- identifiers' whose- timestamps correspond tothe biniiabel. During each time slot, the current time CT, 
advances to point to the next tan. The VC identifiers linked to bins which have been passed by CT are all ready for sery- 
ice. For each priority level, a ready list of read^ bins is constructed/VGs are then served from the ready list in round- 
robin fashion, according to priority level. 

TO " 

Providing Multi-class Quality-of Service 

[0082] In drd4r to pirovide qualityK)f-service (QoS). a.connectiori.admission .control (CAC) algorithm is necessary to 
determine whether or not a new VC can be accepted while satisf/ing the iQpS requirement pf.the new VC and the^exis.t-^ 

15 ing VCs. DRC scheduling simplifies the CAC function by providing a direct mapping between the bandwidth required to 
provide QoS to a stream and the rate at which the stream is scheduled for service. In particular, the CAC determines a 
minimum required bandwidth to provide QoS for a traffic stream and this bandwidth Is used as the minimum guaranteed 
rate for the stream under DRC. If the DRC scheduler is able to provide the minimum guaranteed rate, then the QoS for 
the stream will also be satisfied. In DRC. the method of computing the DRC rate depends on whether cells are queued 

20 by dass or by VC. 

[0083] From an'ahalysis point of view, per VC queuing Is preferable; however, from an implementation point of view, 
per VC may not be preferable since It requli'eis a larganumber of queui^s.'The rhain advantage of per VC queuing is that 
each VC is guaranteed a minimum rate. Additionally, in a per VC queuing, all downstream bottlenecks are considered 
in calculating the rate, thus avoiding overflow of downstream buffers. Also, in a per VC queuing there would be no head- 
25 of -line blocking. On the other hand, per class queuing is simpler to implement If only one downstreani bottleneck is con- 
sidered. Notably, such an arrangement requires fewer queues.. , 

[0084] First, queues will be described for the case wherelh a static' weight. \A/j. is assigned to each class by the CAC. 
The value of the weight Wj. determines the share of the free bandwidth that is allocated to class 1. Thereafter a method 
wherein the weights are modified dynamically in the context of a closed-loop control, based on observed QoS will be 
30 disclosed. / r : - • 

Per Class Queuing 

[0085] When cells are queued by class, the traffic stream corresponding to class i Is an aggregate of several Individual 
35 VC Streams, the CAC determines a minimum' required bandv/idth, idencted Mj, needed to satisfy .the cell loss and cell - 
delay requirements. The CAC provides the DRC scheduler with the minimum guaranteed rate, Mj, and also the class 
weight. Wj. The dynamic rate for stream i at a single bottleneck point is computed as 

40 ^'-^ , .-Ri(n) = Mj.-h; . ^' .,E(n). ; (44) 

where Wj Is the weight assigned to stream i and A(n) is the set of 'active' streams over the time interval ((n-1)A,nA). To 
45 simplify notation, we shall Implicitly assume that all computed rates are multiplied by the Indicator function l[o,c](^) to 
ensure that the rates fall In the range [0,C]. Methods for estimating the hunniber (or weighted sunn) of active streams or 
VCs, e.g., the sum . . 

' ' ' ' ' ^ ' • ; . ^ , . . , ~ - 

S(n) = X w j, (45) 

jeA(n) 

in Eq. (43) will be discussed further below. 

[0086] Suppose that we can obtain an estimate for ni(n), the number of active VC streams composing stream i. By 
55 setting w j = n j(n) . the unused bandwidth can be distributed fairly with respect to the Individual VCs. In this case, each 
VC will receive an equal share of the 'excess bandwidth -E(n).- More generally, both, a class weight and the. number of 
active VCs can be taken into account in distributing the excess bandwidth. If denotes the static class weight,- the DRC 
weights would be assigned as vVj(n)-*\j/jnj(n). f j.. : . r _ - . 
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Per VC Queuing 



r0087] " In per VC queuing, for N VC there are N VC queues. Each VC queue belongs to one of K.9la?ses Let C(i).„ 
denoti the dass to which VC i belongs and let C. denote the set of VCs belonging.to "^'f ^^.^Syi'Jf ST^c^a^k ' 
5 mum guaranteed bandwidth M^. which is sufficient to guarantee QoS for.all :VCs;.rT, plas? 

biomes inactive, the unused bandwidth is first made available to VCs belonging tO:Class..cW.. MP to, the . class guar-., 
anteed bandwidth Mn. aiid then to the VCs in the other classes. 'r,- • . • ; - 

[0088] In the ^)er vC paradigm a dynamic rate is computed for VC i as foHows: , :^ . ■. ii • . ; ■ , 

,0- Ri(n) = Mi + E,(i)(n), " (46) 

- ■'. -'It..,; c ,>■>•' ■■■■ ■■ . 

where E^(n) denotes the estimated unused bandwidth for class k at the sampling instant n. We assume that the flew 
* rate at timi n TJnl. of class k cells into the common queue.can.be rtfeasured. -Mso. we. assurBe,tliat,,t is poss.ble,to. 
cour^t the nurnber.-Qk(n)'. of class k cellsin the common queue, aftimen. The common. qu^^ length is given by . 
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[0089] The excess'baridwidth^at the bottleneck point, denoted E(n). is estimated..as, a.fiJnctio9,,9f.,th,eaqgregate flw 
rat4, F{n)i and the common queue length, Q(n). It can be computed using theiirybrid PDcQntrplter discussed earlier: 

:\ , E(n+l)=E(n>,.go(F(nX^yoC)^d;^^^^^ 

E(n+1) =E(n) - a'o(Q(n) - Qo) = a',(Q(n-l)-Qo) 
end if ' 

35 [0090] The mihimum guaranteed class bandwidthi-Mk^is determined by the CAC. The dyrianriic Vate tor class k is^m- ' 
puted as:"' ' '■■ ■-■ - r '■ 'V-y ^-y..^ ., .... - . • --r,'^ i.-.,™^ 

■• - '^ y -■ ■ ' •'^ '^-r ^ - 

R^(n)=MK+Wk"k(")^(")- ^ ' 

40 Where w, is the weight tor Class k and nk(n) isan estimate-foc.the number of active VCs belonging to^^^ 

[00911 The dynamic class rate Rk(n) represents the bMndwIdth available to VCs in class k. For VCs .n class k. the 
[Tsi bJndwl^th is computed with'respect to R.(n). The unUsed bandwidth for class k, E,(n). can be computed using 

the hybrid PD controller as follows: 

' . . ' ifFk(n)<Uo^''lRk(n)tiien:' V ^ . -. . ".■: ' .,7 . 

Ek(n+l)=Ek(n)-ao(Fk(n)-Uo<''^Rk(n))-a,(F,(n-l)-Uo^''^Rk(n-l)) - = • 

else 

E,(n+l)=E,(n).a'o(Qk(n)-Qo^'W^(QK(n-l)-Qo^'^ 

• . > ^ end if - •: i '/ 

Here Un^*'^ and Qn^'^Ure. respectively, -the target utilization and -.v - ^ ;^.u a*-.* 

[0092] Thus, the above novel per VC queuing accomplishes a two-tier .distribution of the unused bandwidth. A first 
distribution according to class and a second distribution according to VCs within the class; . . , , 
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[0093] • ''ln''equati6ri (46) it is implied 'thai E^(j)(n) is distri^ evenly among the active streams within .the dass. How- 
ever, continuing withlhe DRC thernet dne may elect to have variable distribution. This can-be easily accomplished* by 
introducing a weight factor, e.g., assigned to each VC within the class. Thus, the rate would be computed as: 

. , . _ Rj(n) = M: + (t)iEc(j^(n). 

Closed-loop Quality-ofiService Control ' 

[0094] In the DRC scheme, the excess bandwidth, E(n), is computed via a feedback control loop. This excess band- 
width is distributed among the streams be means of the weights. Wj. In the previous section, the weights were assumed 
to be static, and chosen by the CAC. However, the most general form of ORG allows the.weights toi^e time-varying (cf. 
Eq. (2)). In the fdjiowihg, we develop a schi&me for'prdviding closed-loop QoSxontrol in.conjunction.with DRG, vyhereby 
the weights are adjusted based on the observed QoS of the streanis. For the purposes of this discussion, we will 
assume per dass queuing, although the same methcK^ology applies under per VC queuing as well. , r 

Concept - ' ^ : .. ^ • ■ t < 



[009i5] As discussed jn tfie previous "section, s CAC function is necessary to guarantee QoS. However, since the traffic, 
streams are inadequately characterized, the C!aC rhay overestimate or underestimate the amount of bandwidth which 

20' should be allocated to a stream in order to guarantee QoS. The only way to determine whether QoS guarantees are 
being satisfied is to measure, in some way. the QoS experienced by an individual stream. If the measured QoS falls 
below the target QpS, more bandwidth should be allocated to the stream. Conversely, if the QoS for a stream exceeds 
the target QoS, It may be possible to fake sorhe baindwidth away frorn the stream and make it available to other streams 
which are in greater need of the'biandwidth. " 

25 [0096] A dynamic or 'measui^emeiit-based CAC (fcf: G. Ramamiirthy cited above^ and S - Janni n , R Danzig, S. -Shenkerr 
and L Zhang. A Meksui'ement-Based Admission Coiifr^ Algorithm for Integrated Service Packet Networks, IEEE/ ACM 
Trans, on Networking, vol 9, pp. 56-70; February 1997) adjusts the bandwidth assigned -to a stream base on traffic 
measurements. Measurements take place over a relatively long time interval (e.g., an order of magnitude less than call 
holding times). Then ccTrr^ibns are made to the CAC bandwidth allocation. For example, it may be possible to obtain 

30 reasonable estihriate 6f cisll loss probability over a tinie interval: A fui'ther challenge' is hovy to determine how much 
bandwidth should be added to a connection suffering QoS degradation. The dosed -loop QoS control method disclosed ; 
in this section opeirates on a shorter tirfie^scale compared with the dynamic CAC. e.g.. onjthe same order as the 
dynamic rate compiilations. The closed-loop cbntrol can make short-term corrections to alleviate errors made by the 
CAC. The closed-loop control can also work well in conjunction with'a' dynamic CAC. 

35 [0097] The basic jdea is to assign the weight, Wj(t), proportional to the deviation between the observed QoS at time t 
and a target QpS measCire for streaml. Let Qj<t) be the 6&sefved'Q6S for stream i in the first-stage queue i at time t.- 
For example, qj(t) cpiild represent ai measure of the cell dfelay e^^perienced at tinne t. Let qj* represent the target QoS for 
queue i .The hormalized deviation of the observed QoS from the target QoS is given by: = ; 



Since the 

-'^ - ' .■_,qi(t)/,qt....- 



'i (49) 



weights ' " ' ' ^- '■ - -: - ■ : - 

should always be positive, we assign them as follows: , 

so . Wj(t) = — . (50) 

[0098] With this assignment of weights, strearhs which are further away from their QoS targets receive greater shares 
of the excess bandwidth. Conversely, streams which are meeting or exceeding their QoS targets will receive proportion- 
55 ately smaller shares of the excess bandwidth. ' ^ 

[0(^9] The closed-loop QoS control attempts to minimize the worst-case deviation of a given stream from its target 
QoS. Witti the. assignment of weights according to Eq; (50). each stream effectively makes a bid for the excess band- 
width based ori the ratio of its perceived QoS to its target QoS. In this way; a stream suffering from poor QoS automat- 
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ically takes-more of the available excess bandwidth compared with a stream whlch is meeting or exce^ing.its.target. , 
QoS. Closed-loop QoS'can make short-term corr;eGtions.to errors iij bandwidth allocation macle by .tf^e.CAg.. ,. , . . 

QoS Measurement 

10100] In a real implementation, the QoS of a stream must be measured over a time interval. Updates toj^e weighte 
take p ace at discrete times. A reasonable scheme is to re-compute the weights directly, either before pr after the co^- . 
putaion of the excess bandwidth. As an example, the average cell delay. D(n). in the first stage queue ^ stream can 
be rtieasured over the time interval (nA (n + 1)A). Average queue length is also a.relatively simple weighing. function . 

roiOl]*^On^the Sher hand, it is very difficult to estimate cell loss provability qyer a relatively shprt period, ohinie. This 
QoS metric can only be estimated over a longer tin^^-interval.^A dynamic.CA&giight be able to estimate.cell 'ofs Prob- 
* ability and to allocate sufficient bandwidth to correct t,l?.e. eelt Ipss probatJility for the ne)ft measurerrient interval, based , 
on observations of the ti-affic oven the cun-ent.i.^teayal.. -i::-, •, ■ . •• . 

J5 

Congestion Control via Dynamic Rate Control 

[0102] In.this section, we discuss tiow-DRC cpi b^...usecl to corrtrol congestion at bQlttenecks within a multi-stag^ 
switch. We then discuss how.DRC can be exteoded beyond the Ipcal switch to provide hop-by-.hop.,conge?t.on control.^ 
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Input-Output Buffered I Switch «. - . .. u, .. „ , ';: .■ • .■ y i- •■;!.■ •, v ;•^ 



[0103].- The prior art. weighted.fair share schedulers distribute bai^width weighted.with respect to a singlebottleneck. 
Minimum rate guarantees can be provided with respect to this bottleneck. However, if therejs a^s^corxJ, downstream 
bottleneck., the prior Srt weighted fair, share :scheduler,.rpay.,nqt,be able.tp provid^. bap^dwdth guarantees. Thj input 
streams &the. second-stage bottleneck may,.9.cigi.nate from differept^bottleneck.po.nts in the f-rst^t^ge, If thesej«-st 
stage bottlenecks arescheduled independently by weighted^fair share schedulers, congestion at the corpmpn second- 
stage bottleneck.-may arise, resulting in the lo^ of-rate guarantee.,, - .. 

[01 04] . Figure 9 shows an example of a hypothetical N -x N input-output bufferaJ switch.. Input, arid °Mtput.m<giJes 
iml > imn^and Om1 - OMN. respectfuHy.are connected to;g coce svyitching element 1 01 hayjng ^ central hjph speed bus 
120 (e.g:, a time-division muftiplexed bys). Each input ^oodule has a scheduler, whicfi sch,aJule^s c?l^ ^ be transmmg 
over the bus 120: Each output module coiisistg of buffers-PTV RTN which^ 
the line speed. When the output buffer occyp^y:reaches,,a,.ce.itain threshold, a signal ^|?broa^cg to all '"Put rpod- 
ules. The signal causes all input modules to,thrpttle,the,f lQW,^)f,traffic to the given output nnodule. This prevents buffer 
35 overflow at the-output nradule. . ^-r • r.n;-.:-.c , : ^ , ^ j.^ V.,^ r.f,*r..it 

[0105] Consider two streams. 5, and Sg. originating, from:different. input- modules and. destine^ *° -?'"•L^t• 
module. Assume thatboth streams are coqti.)uous,li«.ba§ljl^^^^ 

share schedulers. Since the schedulers.are v.Qrk cpnservjr.g,th^^.ut.put^ cell rate from. each '"P"* "^<^^'^r'''!^f . 
to the line rate C. The output module buffer level will eventually exceed the backpressure threshold. The bacl^ressure 
signal will throttle both input modules until the buffer occupancy at the output module falls below the stop threshold Jhe 
throughput received by each stream will be 0.5C. With weighted fair schedulers, it is not possible ^cjieve differer^t 
throughputs for the two streams. This is because the schedulers are work conserving with respect to the first stage bot- 

[0106] On the other hand, if DRC schedulers are employed at the input modules, the output streams from the Input 
modules can be shaped to different rates. For example, suppose M, = 0.1 C and Mg = 0.8C. Then the excess bandwidtti 
at the output module bottleneck is 0.1C. H this excess bandwidth is distributed evenly between the two streams, the 
throughputs for the two streams will be Ri = CISC and Rg = 0.85C... respectively. ..... „ , = .- ■• , , 

Hop-by- Hop Dynamic Rate Control 

[0107] Dynamic Rate Control can be extended beyond the local switch if cells are queued per If the downstream 
switch is able to transmit rate information on a per VC basis, e.g. . through resource management (RM) cells. The down- 
stream.rate can be used by the local DRC scheduler to.sqheduleyCsas,^^^ 

55 R = min(R|o<^|, R^lownslreaiTi)- ,. .-• • i 

[0108] " While the invention has beeridescribed-with refprence;to specific it vvill be appreciat^ 

by those skilled in the art that numerous variations. . modificatior)^. and embodiments are possible, and accordingly, all 
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such variations, modifications, -and embodiments are to be regarded as being within the spirit and scope of the inven- 
tion. 

[0109] It should be appreciated that while the above disclosure is provided in terms of scheduling cells in ap ATM 
switch, the inventive dynamic rate control (DRC) can be inrplemented for scheduling data packets in packet switching. 
5 For example. Equations 39 and 40 can be easily modified to account for the packet's length as follows: 

TSi = max^^CT,TSi+L/Ri(n)K (39') 

* " . •■-'ic- - ^ * ^: TSj =^TSj +:URi{n):.^: ■ r - r. (40'). 

wherem L represents the length of the packet at the-head of 'the queue being sc^ \ . t.* • 

15 [0110] Although the above-mentioned description has been made about the switch which has input and output buff- 
ers, the present invention is not restricted to this switch but is also readily applicable to an ATM switch which has a com- 
mon buffer. In this case, each queue to be scheduled is k>gically formed in the common buffer. In addition. the minimum 
guaranteed rate and the dynamic rate may be selected as the shaping rate. ' . '* 

[0111] As mentioned above, the present invention is featured by determining each rate of .the queues on the basis of 
20 the two schedule factbrs, the rriinimurh guaranteed rate and the excess bandwidth which are fixed and variable factors, 
respedtively. As^a result! the schedijler according to the present invention can respond to services based on the various 
QoS requirements and can effectively shape the rate on occurrence of a downstream bottleneck. 

Claims 

25 " T ' ■ • . ' ... ^ 

1 . A method of rate-based cell scheduling of a plurality of cells arriving at an ATM switch having a plurality of queues, 
comprising: * " ' *• - • ' ^ :■ . . •^ ^ - . . 

directing each one of said plurality of cells to a respective queue; 
30 assigning a respective minimum rate guarantee for each of said queues; - : 

assigning a respective excess rate share for each of said queues; 

estimating excess bandwidth bri a downstream link; ' ^ \ - ? . * 

transmitting said cell from said queues according to the respective minimum rate guarantee, while distributing 

the excess bandwidth to said queues according to said excess rate share. 

2. A method of rate^based scheduling at an ATM switch haviririg a jplurality of input queues, comprising the steps of: 

assigning a minimum guaranteed rate for each of said queues; 

computing a variable rate for each of said queues; I ' ' : . • . - h\ - - . :. 

40 shaping each stream arriving at each of said queues according to the respective minimum guaranteed rate and 

the respective variable rate. • . . , . - 

3. The method of claim 2, wherein said ATM switch further conrprises a plurality of output buffers, said method further 
comprisfng the' steps of:' ' ^ * ' ' ' ' v.r- i:-^ , v-^*; 

45 

monitoring the level of each of said buffers and. when^orte of said buffers reaches' a^ predetermined level, gen- 
erating a shape signal identifying said one buffer; 

scheduling said queues according to a dynamic rate composed of the minimum guaranteed rate plus the vari- 
able rate when said shape signal is not generated and scheduling said queues accoi-ding to said minimum rate 
so when said shape signal is generated. 

4. A method of rate-based cell scheduling of a plurality of cell streams, comprising the steps of: 

assigning a minimum rate for each of said plui^iity of cell streams;' . - 

55 calculating a dynamic rate for each of said cell str earns, said dynamic rate comprising a product of an assigned 

weight and an estimated excess bandwidth at a downstream bottleneck; and 
_ adding each dynamic rate to a corresponding minimum rate. . > 



2 b 

BNSOOCID: <EP 0901301A2J_> 



EP O 901: 301 A2 f 

5. Themethod^of claim 4/ wherein the excess bandwidth is estim^^^ . , ^' 

6. fhemethodof claim 4, wherein said assigned weight is static. . * ^ ■ 
5 7. The method of claim 4. wherein said assigned weight is dynamic . ■ • i. . ' - 

8. A method for shaping transmission rate of a cell stream arriving at a buffer, comprising the steps of: 

monitoring the level of said buffer; * < *^ 

10 ' when the level of said buffer reaches a first predetermined threshold reducing the transmission rate of said 

stream to a preassigned minimum rate; and 
• ' when the level of said buffer reaches a second predetermined threshold halting transmission of said stream. 

9. A method for queuing a plurality of vintu^l chann^ls in an ATM switch having a plurality of Jnput buffers, comprising 
15 the steps of: ^ - "-v.- - - u .iv:: > t. - ^ . ^ ' * . : . •-..••-■!( 

assigning an input bafferfor each of said virtual channels; s ,^ 
assigning a minimum guaranteed ratQ for eacK-.of:said;buffers; , : ' • ^ * • 

. assigning a weight for each of said buffers; - - =: - i j- . • :^ -v- ^ ^ \ 

20 . calculating a^dynamic rate for each of said.buffers; saiddynamic rate comprising the minimum.guaranteed. rat^ 

- plu$ a portion of an unused-bandwidth of said switch, said ^portion being prppprtionaL to the. assigned vveight; 

shaping transmissions from said buffers according to the dynamic rate. 
25 1 0. The method of claim 9. wherein each of said buffer is assigned to only one virtual channel. 

1 1 . The method of claim 9. wherein each of said buffers is assigned to a plurality of virtual channels having similar qual- 
ity of service requirements, further comprising the step of: 

distributing the dynamic rate of each buff er:toJts. respective active vt . ..^ r . 

1 2, The method of claim 1 1 . wherein said dynamic rate is distributed evenly ampng thefespectiye active virtual chan- 



30 



nels. 

35 1 3 The method of claim 1 1 . wherein each of said virtual channels is assigned' a secondary weight determined for each 
■ class of the service requirements and. whecein the:<iynamic rate-is distributed ta the resp^tive. virtual chamels 
according to the respective secondary weight. 

14. A method for controlling overload in a buffer, comprising: . - . . *v , r;- ?. 

monitoring a load levelin said buffer; - : f / - 'i.u^« ♦ 

when said load level reaches a first threshold, generating a shape signal to cause input to said buffer to be 
reduced to a minimum, level; , - ..... - i.i :«^, 

when said load level reaches a second threshold, generating a stop signal to halt any input to said buffer. . 

45 

15. The method of claim 14, further comprising the Steps of; f, - - . t - ^ . ; . k ^ 

. -itestimateanunused.bandwidth available on isaid buffer; . - ^ 

.generating a signal indicating said estimate. - ^ • : -.^ i • : , f .t:i 

16. A method of rate-based scheduling of a plurality of data packets arriving at a switch having a plurality of queues, 
comprising: / - • ! -.r . , ' . • *: - • - . i - - v 

directing each one of said plurality of packets to a. respective queue; , 
55 - assigning a respective minimurnrate.guarantee for .each pf-said queues: ^ 

assigning a respective excess rate share for each pf said .queues; , ^ , ^ 

estimating excess bandwidth on a downstream Ijnk; - . ... ^ , . , ^ . ^ 
transmitting said packet from said queues according to the respective minimum rate guarantee, while distrib- 
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uting the excess bandwidth to said queues according to said excess rate share. ■ -.. . ^ . . 

17. A Wiethod of 'ratef^ased scheduling at aiswitch having^a plu^ . . 

5 assigning a minimum guaranteed rate for each of said queues; 

computing a variable rate for each of said queues; 

shaping each packet stream arriving at eadi of said queues according to the respective minimum guaranteed 
rate and the respective variable rate. 

70 18. A method of scheduling a plurality of queues in a switch which has a plurality of input ports and a plurality of output 
ports and which is operable to switch between the input and the output ports, the method comprising the steps of: 



calculating a first schedule factor predetermined for each of the queues; 

calculating a second factor which is relatively determined in dependency upon a relationship between each 
75 queue and the remaining queues; and 

determining whether scheduling operation of each queue is carried out only by the first schedule factor or by 
both the first and the second schedule factors. 

19. A method as claimed in claim 18. wherein the first schedule factor is specified by a minimum guaranteed rate for 
20 each of said queues while the second schedule factor Is specified by a variable share rate which is determined by 
the relationship of rates between each queue and the remaining queues. 



20. A method as claimed in claim 19, wherein the variable share rate which is given as the second schedule factor 
depends on a weight allocated to each queue. 

25 

21. A method as claimed in claim 19. wherein the variable share rate which is given as the second schedule factor Is 
determined by detecting a rate of a common queue which is common to the respective queues, by determining a 
minimum guaranteed rate of the queues using the common queue, by adding the rate of the common rate to the 
minimum guaranteed rate to obtain a sum rate between the rate of the common rate and the minimum guaranteed 

30 rate, by calculating an unused rate of the common queue on the basis of the sum rate, and by distributing the 
unused rate. 

22. A method as claimed in claim 21 , wherein the unused rate is determined in relation to a weight predetermined by 
each of the queues: 

35 

23. A method as claimed in claim 21 . wherein the switch is specified by an ATM switch while the weight is dynamically 
determined on the basis of service classes related to connection admission control carried out in the ATM switch 
and current active queues. 

40 24. A method as claimed in claim 23, wherein the unused rate is calculated by the use of a target utilization in the com- 
mon queue and a total flow rate in the common queue. 

25. A method as claimed in claim 23, wherein the unused rate is calculated by the use of a relationship between a tar- 
get queue length of the common queue and a current queue length. 

45 

26. A method as claimed in claim 18, wherein the switch is formed by an ATM switch which has at least one of input 
and output buffers; 

the scheduling operation being carried out in relation to the queues included in each buffer. 
so 27. A method as claimed in claim 1 8, wherein the switch is formed by an ATM switch which has a common buffer; 
the scheduling operation being carried out in relation to the queues included in the common buffer. 
28. A method of rate-based scheduling at an ATM switch having a plurality of input queues, comprising the steps of: 

55 ' 

assigning a minimum guaranteed rate for each of said queues; 
computing a dynamic rate for each of said queues; and 

selectively shaping each stream arriving at each of said queues according to the minimum guaranteed rate and 
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the dynamic rate. 



29. A method as tiaimed^in-daim-28. whereinthe dynamic* rate-is determined^by the cninimum. guaranteed rate and a 
variable rate computed for each of said queues. 
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